
United States Patent and Trademark Office 



UNITED STATES DEPARTMENT OF COMMERCE 
United States Patent and Trademark Office 

Address: COMMISSIONER FOR PATENTS 
P.O. Box 1450 

Alexandria, Virginia 22313-1450 
www.uspto.gov 



APPLICATION NO. 



FILING DATE 



FIRST NAMED INVENTOR 



ATTORNEY DOCKET NO. 



CONFIRMATION NO. 



09/507,526 



02/18/2000 



46069 7590 1 1/29/2005 

F. CHAU & ASSOCIATES, LLC 
130 WOODBURY ROAD 
WOODBURY, NY 11797 



Stephane H. Maes 



Y0999-I78 (8728-306) 



7963 



EXAMINER 



HO, ANDY 



ART UNIT 



PAPER NUMBER 



2194 

DATE MAILED: 1 1/29/2005 



Please find below and/or attached an Office communication concerning this application or proceeding. 



PTO-90C (Rev. 10/03) 



Office Action Summary 


Application No. 

09/507,526 


Applicant(s) 

MAES, STEPHANE H. 


Examiner 

Andy Ho 


Art Unit 

2194 





- The MAILING DATE of this communication appears on the cover sheet with the correspondence address 



Period for Reply 

A SHORTENED STATUTORY PERIOD FOR REPLY IS SET TO EXPIRE 3 MONTH(S) FROM 
THE MAILING DATE OF THIS COMMUNICATION. 

- Extensions of time may be available under the provisions of 37 CFR 1.136(a). In no event, however, may a reply be timely filed 
after SIX (6) MONTHS from the mailing date of this communication. 

- If the period for reply specified above is less than thirty (30) days, a reply within the statutory minimum of thirty (30) days will be considered timely. 

• If NO period for reply is specified above, the maximum statutory period will apply and will expire SIX (6) MONTHS from the mailing date of this communication. 

- Failure to reply within the set or extended period for reply will, by statute, cause the application to become ABANDONED (35 U.S.C. § 133). 

- Any reply received by the Office later than three months after the mailing date of this communication, even if timely filed, may reduce any 
earned patent term adjustment. See 37 CFR 1.704(b). 

Status 

1)13 Responsive to communication(s) filed on 02 September 2005 . 
2a)l3 This action is FINAL. 2b)D This action is non-final. 

3) D Since this application is in condition for allowance except for formal matters, prosecution as to the merits is 

closed in accordance with the practice under Ex parte Quayle, 1935 CD. 11, 453 O.G. 213. 
Disposition of Claims 

4) S Claim(s) 1-37 is/are pending in the application. 

4a) Of the above claim(s) is/are withdrawn from consideration. 

5) D Claim(s) is/are allowed. 

6) E3 Claim(s) 7-37 is/are rejected. 

7) D Claim(s) is/are objected to. 

8) D Claim(s) are subject to restriction and/or election requirement. 

Application Papers 

9) D The specification is objected to by the Examiner. 

10) D The drawing(s) filed on is/are: a)D accepted or b)D objected to by the Examiner. 

Applicant may not request that any objection to the drawing(s) be held in abeyance. See 37 CFR 1.85(a). 

1 1) D The proposed drawing correction filed on is: a)D approved b)D disapproved by the Examiner. 

If approved, corrected drawings are required in reply to this Office action. 

12) D The oath or declaration is objected to by the Examiner. 
Priority under 35 U.S.C. §§119 and 120 

13) D Acknowledgment is made of a claim for foreign priority under 35 U.S.C. § 1 19(a)-(d) or (f). 

a)DAII b)D Some*c)D None of: 

1 .□ Certified copies of the priority documents have been received. 

2. D Certified copies of the priority documents have been received in Application No. . 

3. D Copies of the certified copies of the priority documents have been received in this National Stage 

application from the International Bureau (PCT Rule 17.2(a)). 
* See the attached detailed Office action for a list of the certified copies not received. 

14) D Acknowledgment is made of a claim for domestic priority under 35 U.S.C. § 1 19(e) (to a provisional application). 

a) □ The translation of the foreign language provisional application has been received. 

15) D Acknowledgment is made of a claim for domestic priority under 35 U.S.C. §§ 120 and/or 121. 

Attachment(s) 

1) □ Notice of References Cited (PTO-892) 4) □ Interview Summary (PTO-413) Paper No(s). . 

2) □ Notice of Draftsperson's Patent Drawing Review (PTO-948) 5) □ Notice of Informal Patent Application (PTO-1 52) 

3) □ Information Disclosure Statement(s) (PTO-1449) Paper No(s) 3/7/2005 . 6) □ Other: 



U.S. Patent and Trademark Office 
PTO-326 (Rev. 04-01) 



Office Action Summary 



Part of Paper No. 



Application/Control Number: 09/507,526 Page 2 

Art Unit: 2194 

DETAILED ACTION 

1 . This action is in response to the response filed 9/2/2005. 

2. Claims 1-37 have been examined and are pending in the application. 



Claim Rejections - 35 USC § 103 

The following is a quotation of 35 U.S.C. 103(a) which forms the basis for all 
obviousness rejections set forth in this Office action: 

(a) A patent may not be obtained though the invention is not identically disclosed or described as set 
forth in section 102 of this title, if the differences between the subject matter sought to be patented and 
the prior art are such that the subject matter as a whole would have been obvious at the time the 
invention was made to a person having ordinary skill in the art to which said subject matter pertains. 
Patentability shall not be negatived by the manner in which the invention was made. 



3. Claims 1-15 and 19-37 are rejected under 35 U.S.C. 103(a) as being 
unpatentable over Hashimoto U.S Patent No. 6,119,147. 

As to claim 1, Hashimoto teaches a system comprising: 
a program storage device (a computer, line 38 column 3) that stores a multi- 
modal application (combination of SRS 1 and application programs 2, Fig. 6), the multi- 
modal application comprising at least a first and a second mode process (inputs and 
outputs from the application programs, lines 23-25 column 41 ; speech synthesis unit 
operated as an independent process, lines 37-38 column 42) that enables user 
interaction with the application in a first modality and second modality (the system 
allows the user to interact with the system via text data or speech data, lines 50-63 
column 49; Figs. 17, 66, and 69-72); 
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a program execution system (speech recognition interface system, lines 11-12 
column 10) that executes the multi-modal application and synchronizes the first and 
second mode processes while a user interacts with the multi-modal application (lines 
12-37 column 10) wherein the program execution system comprises: 

a multi-modal shell (1 1 , Fig. 56) that manages information exchanges between 
the processes (controls by exchanging the messages transmitted from the speech unit 
to the application program, lines 38-41 column 10) to enable a synchronized multi 
modal interaction with the application (the system allows the user to interact with the 
system via text data or speech data, lines 50-63 column 49; Figs. 17, 66, and 69-72) 
wherein user interaction in one modality results in execution of corresponding 
commands in both the first and second mode processes (lines 38-63 column 49); 

the processes register their respective active commands and corresponding 
actions (register the recognition vocabularies and the appropriate actions that response 
to these vocabularies, line 48 column 10 to line 10 column 1 1) with the multi model shell 
(11, Fig. 56). 

Hashimoto does not explicitly teach an API. However, Hashimoto teaches that 
each application program includes a message I/O unit (line 9-29 column 12) wherein all 
of the application program interactions with the speech recognition system are handled 
by this message I/O unit. Therefore one of ordinary skill in the art would conclude that 
this message I/O unit could be used as an API since it provides the interface for each 
application program to interact with the speech recognition system. 
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As to claim 2, Hashimoto as modified further teaches a registry having a 
registration table (program management table, line 58 column 10), managed by the 
multi-modal shell (1 1 , Fig. 56), that comprises a list of each of the registered commands 
and corresponding synchronized actions (the recognition vocabularies and the 
appropriate actions that response to these vocabularies, line 48 column 10 to line 10 
column 1 1 ) that results in both the first and second mode processes upon execution of 
a registered command (once the speech input "Finish" is transmitted, both of the 
application program can be finished by this single speech input, lines 5-8 column 27) by 
one of the first and second mode processes (inputs and outputs from the application 
programs, lines 23-25 column 41). 

As to claim 3, Hashimoto as modified further teaches the multi-modal 
application comprises a first mono-mode application (speech synthesis unit, line 37 
column 42) for the first mode process and a second mono-mode application (application 
program, lines 41-42 column 42) for the second mode process, wherein the multi-modal 
shell (11, Fig. 56) manages and synchronizes information exchanges (message 
exchanges using the process communication, lines 39-40 column 42) between the first 
(speech synthesis unit, line 37 column 42) and second mono-mode applications 
(application program, lines 41-42 column 42). 

As to claim 4, Hashimoto as modified does not teach devices having user 
interface modalities. However, Hashimoto teaches (lines 51-55 column 12) the 
message system can be implemented as a server and clients system wherein the 
speech recognition unit can act as a server and the applications programs are clients. 
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Therefore one of ordinary skill in the art would conclude that the clients are the devices 
with modalities wherein these devices can register their commands and corresponding 
actions with the server. 

As to claim 5, Hashimoto as modified further teaches the devices multi-modal 
shell are distributed over a network (server and client system, lines 51-55 column 12), 
and wherein the API is implemented using distributed APIs or protocols (byte stream 
type protocol, lines 54-55 column 12). 

As to claim 6, Hashimoto as modified further teaches a mechanism for 
converting a mono-mode application to a multi modal application (application program 5 
is interact with both speech input and keyboard input, Fig. 17). 

As to claim 7, Hashimoto as modified further teaches the mono-mode 
application is a GUI application (user interface of Fig. 18), the mechanism (11, Fig. 56) 
provides speech enablement (controls by exchanging the messages transmitted from 
the speech unit to the application program, lines 38-41 column 1 0) of the GUI 
application (user interface of Fig. 18) by registering the active commands of the GUI 
application and building a grammar for the registered commands to support the 
commands in a speech modality (register the recognition vocabularies and the 
appropriate actions that response to these vocabularies, line 48 column 10 to line 10 
column 11). 

As to claim 8, Hashimoto as modified further teaches a mechanism for building 
a multi-modal application (application program 5 is interact with both speech input and 
keyboard input, Fig. 17). 
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As to claim 9, Hashimoto as modified further teaches the mechanism (1 1 , Fig. 
56) is used for directly programming the registry by building a registration table 
(program management table, line 58 column 10) having user-defined commands and 
corresponding actions (the recognition vocabularies and the appropriate actions that 
response to these vocabularies, line 48 column 10 to line 10 column 1 1 ) for each of the 
modalities of the multi-modal application (1A, Fig. 56). 

As to claim 10, Hashimoto as modified further teaches that the system is 
implemented on personal computers, workstations... (lines 8-12 column 1). Hashimoto 
does not explicitly disclose an operating system. "Official Notice" is taken that both the 
concept and advantage of providing for an operating system is well known and 
expected in the art. It would have been obvious to include an operating system into the 
system of Hashimoto because it would provide the execution space for the system. 

As to claim 11, Hashimoto as modified further teaches the system is distributed 
over a network (server and client system, lines 51-55 column 12). 

As to claim 12, Hashimoto as modified further teaches the multi-modal 
application (1A, Fig. 56) is a multi-modal browser (a mail browser, Fig. 27), comprising 
first and second browser applications (multiple application programs 2, Fig. 7). 

As to claim 13, Hashimoto as modified further teaches the first browser is GUI 
(interface of Fig. 27) and the second browser is speech (the user can command to open 
the mail by saying "yes", lines 54-60 column 25). 

As to claim 14, Hashimoto as modified further teaches the multi-modal shell (11, 
Fig. 56) processes the multi-modal application (recognition result, line 16 column 25) to 
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send modality specific presentation information (speech result from the speech 
recognition system, lines 14-17 column 25) to the respective browsers (a mail browser, 
Fig. 27). 

As to claim 15, Hashimoto as modified further teaches the multi-modal 
application is authored using a modality-independent description and wherein the multi- 
modal shell generates (recognition result, line 16 column 25) the modality-specific 
presentation information (speech result from the speech recognition system, lines 14-17 
column 25) from the modality-independent description (input speech, line 12 column 
25). 

As to claim 19, Hashimoto teaches a method comprising: 
activating a multi-modal application (combination of SRS 1 and application 
programs 2, Fig. 6) comprising at least a first mode process (inputs and outputs from 
the application programs, lines 23-25 column 41 ; speech synthesis unit operated as an 
independent process, lines 37-38 column 42) that enables user interaction with the 
application in a first modality (the system allows the user to interact with the system via 
text data or speech data, lines 50-63 column 49; Figs. 17, 66, and 69-72) and a second 
mode process (inputs and outputs from the application programs, lines 23-25 column 
41 ; speech synthesis unit operated as an independent process, lines 37-38 column 42) 
that enables user interaction with the application in a second modality (the system 
allows the user to interact with the system via text data or speech data, lines 50-63 
column 49; Figs. 17, 66, and 69-72); 
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receiving a command (speech input, line 14 column 41) in a first modality 
(speech recognition interface system, line 13 column 41); 

triggering an action (speech output, line 19 column 41 ) in the first mode process 
based on the received command (speech recognition interface system, line 13 column 
41) and triggering a corresponding action (the mail is opened when the user saying 
"yes", lines 54-60 column 25) by the second mode process (application program, lines 
41-42 column 42); 

updating application state associated with the second mode process (update the 
program management data in the program management according to an internal state 
of each application program, lines 35-39 column 76). 

Hashimoto does not explitcitly teach updating application state associated with 
the first mode process. However, Hashimoto teaches (lines 37-46 column 14) the 
speech recognition system also has its own internal state; and when there is a change 
in this internal state, the application program will get a message notifies about this 
change. Therefore one of ordinary skill in the art would conclude that in a certain time, 
the internal state of the speech recognition system is updated; therefore produces the 
state change. 

As to claim 20, Hashimoto as modified further teaches registering active 
commands associated with the first and second mode processes (the recognition 
vocabularies and the appropriate actions that response to these vocabularies, line 48 
column 10 to line 10 column 11); associating, with each registered command of the 
mode processes, an action on one mode process and a corresponding action on the 
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other mode process (once the speech input "Finish" is transmitted, both of the 
application program can be finished by this single speech input, lines 5-8 column 27). 

As to claim 21, Hashimoto as modified further teaches building a command to 
action registration table (program management table, line 58 column 10) based on the 
registered commands and actions (the recognition vocabularies and the appropriate 
actions that response to these vocabularies, line 48 column 10 to line 10 column 11). 

As to claim 22, Hashimoto as modified further teaches the registration table 
(program management table, line 58 column 10) is built by a multi-modal shell (11, Fig. 
56) . Hashimoto does not explitcitly teach an API. However, Hashimoto teaches that 
each application program includes a message I/O unit (line 9-29 column 12) wherein all 
of the application program interactions with the speech recognition system are handled 
by this message I/O unit. Therefore one of ordinary skill in the art would conclude that 
this message I/O unit could be used as an API since it provides the interface for each 
application program to interact with the speech recognition system. 

As to claim 23, Hashimoto as modified further teaches looking up the received 
command (the recognition vocabulary lists, lines 59-60 column 10) in the registration 
table (program management table, line 58 column 10); and executing the actions 
associated with the received command on the first and second mode processes (once 
the speech input "Finish" is transmitted, both of the application program can be finished 
by this single speech input, lines 5-8 column 27). 

As to claim 24, Hashimoto as modified further teaches registering a callback 
handle for each of the registered commands to notify the first and second mode 
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processes of completion of the actions corresponding to the registered commands (after 
the dictionary production is completed, notifies this act to the data acquisition unit 8 by a 
message indicating the completion of the dictionary production, lines 3-5 column 33). 

As to claim 25, Hashimoto as modified further teaches executing the callback 
handle associated with the received command to trigger callback actions on the mode 
processes (after the dictionary production is completed, notifies this act to the data 
acquisition unit 8 by a message indicating the completion of the dictionary production, 
lines 3-5 column 33). 

As to claim 26, Hashimoto as modified further teaches executing first thread 
associated with the received command ("Finish", line 6 column 27); triggering a 
corresponding second thread to initiate the corresponding action on the second mode 
process (once the speech input "Finish" is transmitted, both of the application program 
can be finished by this single speech input, lines 5-8 column 27). 

As to claim 27, Hashimoto as modified further teaches the threads are applets 
("Yes" and "No" icons in Fig. 27). 

As to claim 28, Hashimoto as modified further teaches the threads communicate 
via socket connections (server and client system (lines 51-55 column 12) with a mail 
browser, Fig. 27)). 

As to claims 29-37, they are device claims of claims 19-21 and 23-28, 
respectively. Therefore, they are rejected for the same reasons as claims 19-21 and 
23-28 above. 
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4. Claims 16-18 are rejected under 35 U.S.C. 103(a) as being unpatentable over 
Hashimoto in view of Toomey U.S Patent No. 6,119,147. 

As to claim 16, Hashimoto as modified does not explitcitly teach the multi-modal 
application comprises a combination of declarative markup languages. Toomey 
teaches a multi-model document with a combination of declarative markup languages 
(text discussion, audio commands, graphics, and documents, lines 60-61 column 1). It 
would have been obvious to apply the teachings of Toomey to the system of Hashimoto 
because this document will provide the user with the convenience of interacting with the 
system using the choice of texts or speech commands. 

As to claim 17, Toomey further discloses combining the declarative markup 
languages and synchronization elements to provide tight synchronization (interactions 
are inserted into the multi-modal document at a point that is chronological in the 
meeting to create a synchronous meeting, lines 11-13 column 15). 

As to claim 18, Toomey further discloses separate files for each of the 
declarative markup languages (multiple tracks in the multi-modal document, lines 59-60 
column 1). 

Response to Arguments 

5. Applicant's arguments filed 9/2/2005 have been fully considered but they are 
not persuasive. 

Applicant argued that the claim features "multimodal application comprising first 
and second mode processes" are not addressed (Remarks, third paragraph page 9). In 
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response, these claim limitations are clearly addressed above wherein Hashimoto 
teaches a multi-modal application (combination of SRS 1 and application programs 2, 
Fig. 6), the multi-modal application comprising at least a first and a second mode 
process (inputs and outputs from the application programs, lines 23-25 column 41 ; 
speech synthesis unit operated as an independent process, lines 37-38 column 42) that 
enables user interaction with the application in a first modality and second modality (the 
system allows the user to interact with the system via text data or speech data, lines 50- 
63 column 49; Figs. 17, 66, and 69-72). 

Applicant argued that Hashimoto reference does not teach a multi-modal 
application (Remarks, second complete paragraph page 10). In response, as disclosed 
in the claims rejection above, Hashimoto teaches the system allows the user to interact 
with the application via text data or speech data (lines 50-63 column 49; Figs. 17, 66, 
and 69-72). These interactions are being provided via SRS 1 (Fig. 6) acting as an 
interface of the application. Therefore, the SRS and the application together made up a 
multi-modal application. The reference meets the limitation as claimed. 

Conclusion 

THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time 
policy as set forth in 37 CFR 1 .1 36(a). 

A shortened statutory period for reply to this final action is set to expire THREE 
MONTHS from the mailing date of this action. In the event a first reply is filed within 
TWO MONTHS of the mailing date of this final action and the advisory action is not 
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mailed until after the end of the THREE-MONTH shortened statutory period, then the 
shortened statutory period will expire on the date the advisory action is mailed, and any 
extension fee pursuant to 37 CFR 1 .136(a) will be calculated from the mailing date of 
the advisory action. In no event, however, will the statutory period for reply expire later 
than SIX MONTHS from the mailing date of this final action. 

Any inquiry concerning this communication or earlier communications from the 

examiner should be directed to Andy Ho whose telephone number is (571) 272-3762. 

A voice mail service is also available for this number. The examiner can normally be 

reached on Monday - Friday, 8:30 am - 5:00 pm. 

If attempts to reach the examiner by telephone are unsuccessful, the examiner's 

supervisor, William Thomson can be reached on (571) 272-3718. 

Information regarding the status of an application may be obtained from the 

Patent Application Information Retrieval (PAIM) system. Status information for 

published applications may be obtained from either Private PAIR or 1 Public PAIR. 

Status information for unpublished applications is available through Private PAIR only. 

For more information about the PAIR system, see http://pair-direct.uspto.gov. Should 

you have questions on access to the Private PAIR system, contact the Electronic 

Business Center (EBC) at 866-217-9197 (toll-free). 

Any inquiry of a general nature or relating to the status of this application or 

proceeding should be directed to the receptionist whose telephone number is 571-272- 

2100. 

Any response to this action should be mailed to: 
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Commissioner for Patents 
P.O Box 1450 
Alexandria, VA 22313-1450 
Or fax to: 

• AFTER-FINAL faxes must be signed and sent to (571 ) 273 - 8300. 

• OFFICAL faxes must be signed and sent to (571 ) 273 - 8300. 

• NON OFFICAL faxes should not be signed, please send to (571 ) 273 - 3762 



A.H 

November 22, 2005 




